Information processing apparatus, inspection system, information processing method, and storage medium

ABSTRACT

An information processing apparatus comprises: acquiring a first image including at least a part of an inspection device and a second image including at least a part of a subject; estimating a position of the inspection device based on the first image; estimating position and orientation information for the subject based on the second image; identifying an inspection region of the subject from results of estimating; and receiving a finalization instruction to finalize an inspection result to be recorded or output to an external apparatus based on sequentially acquired inspection results of the subject, and a finalization instruction to finalize inspection region information to be recorded or output to the external apparatus in association with the inspection result, wherein the finalization instruction for the inspection region information is received before the finalization instruction for the inspection result.

BACKGROUND Field

The present disclosure relates to an information processing apparatusused for diagnoses based on medical images in the medical field, aninspection system, an information processing method, and a storagemedium.

Description of the Related Art

In the medical field, doctors make diagnoses by using medical imagescaptured by various modalities (inspection systems). Modalities includeultrasonic diagnostic apparatuses, photoacoustic imaging apparatuses(hereinafter referred to as photoacoustic tomography (PAT) apparatuses),magnetic resonance imaging (MRI) apparatuses, and computer tomographyapparatuses (hereinafter referred to as X-ray computed tomography (CT)apparatuses). Japanese Patent Laid-Open No. 2018-175007 discusses asystem for determining (identifying) which portion of a subject isimage-captured for medical images to be used for diagnoses on theseapparatuses, based on the positional relation between an inspectionsystem and the subject.

More specifically, the system captures inspection images and thenacquires an outer appearance image in which the subject under inspectionand a probe are captured. The system then identifies the positions ofthe subject and the probe based on the acquired outer appearance imageand then automatically calculates an inspection region based on therelation between the two positions.

However, according to the procedures discussed in Japanese PatentLaid-Open No. 2018-175007, the identification of a region calculatedfrom a camera image captured at the same time may fail, or incorrectestimation may occur despite a suitable ultrasonic image having beenacquired. In such a case, the inspector needs to attempt to re-acquirean ultrasonic image and a camera image while suitably bringing theultrasonic probe into contact with the subject again. This results in aburden on both the inspector and the subject and this is an issue.Moreover, there is another issue associated with giving up automaticdata entry using a camera that results in the inspector needing tomanually operate a control panel to enter region information. This is aburden on both the inspector and the subject.

SUMMARY

The present disclosure in its first aspect provides an informationprocessing apparatus comprising: at least one processor; and a memorystoring instructions that, when executed by the at least one processor,cause the at least one processor to: acquire a first image including atleast a part of an inspection device and a second image including atleast a part of a subject, the first and the second images beingcaptured by an imaging apparatus; estimate, as a first estimation, aposition of the inspection device based on the first image; estimate, asa second estimation, position and orientation information for thesubject based on the second image; identify an inspection region of thesubject from a result of the first estimation and a result of the secondestimation; and receive a finalization instruction to finalize aninspection result to be recorded or output to an external apparatusbased on sequentially acquired inspection results of the subject, and afinalization instruction to finalize inspection region information to berecorded or output to the external apparatus in association with theinspection result, wherein the finalization instruction for theinspection region information is received before the finalizationinstruction for the inspection result.

The present disclosure in its second aspect provides an inspectionsystem comprising: an imaging apparatus configured to capture a firstimage including at least a part of an inspection device and a secondimage including at least a part of a subject; at least one processor;and a memory storing instructions that, when executed by the at leastone processor, cause the at least one processor to: inspect a subject;estimate, as a first estimation, a position of the inspection devicebased on the first image; estimate, as a second estimation, position andorientation information for the subject based on the second image;identify an inspection region of the subject from a result of the firstestimation and a result of the second estimation; and receive afinalization instruction to finalize an inspection result to be recordedor output to an external apparatus based on sequentially acquiredinspection results, and a finalization instruction to finalizeinspection region information to be recorded or output to the externalapparatus in association with the inspection result, wherein thefinalization instruction for the inspection region information isreceived before the finalization instruction for the inspection result.

The present disclosure in its third aspect provides an informationprocessing method comprising: acquiring a first image including at leasta part of an inspection device and a second image including at least apart of a subject, the first and the second images being captured by animaging unit; estimating, as first estimating, a position of theinspection device based on the first image; estimating, as secondestimating, position and orientation information for the subject basedon the second image; identifying an inspection region of the subjectfrom a result of the first estimating and a result of the secondestimating; and receiving a finalization instruction to finalize aninspection result to be recorded or output to an external apparatusbased on sequentially acquired inspection results of the subject, and afinalization instruction to finalize inspection region information to berecorded or output to the external apparatus in association with theinspection result, wherein, the finalization instruction for theinspection region information is received before the finalizationinstruction for the inspection result.

The present disclosure in its fourth aspect provides a non-transitorycomputer-readable storage medium storing instructions that, whenexecuted by at least one processor of an information processingapparatus, configure the at least one processor of the informationprocessing apparatus to perform operations comprising: acquire a firstimage including at least a part of an inspection device and a secondimage including at least a part of a subject, the first and the secondimages being captured by an imaging apparatus; estimate, as a firstestimation, a position of the inspection device based on the firstimage; estimate, as a second estimation, position and orientationinformation for the subject based on the second image; identify aninspection region of the subject from a result of the first estimationand a result of the second estimation; and receive a finalizationinstruction to finalize an inspection result to be recorded or output toan external apparatus based on sequentially acquired inspection resultsof the subject, and a finalization instruction to finalize inspectionregion information to be recorded or output to the external apparatus inassociation with the inspection result, wherein the finalizationinstruction for the inspection region information is received before thefinalization instruction for the inspection result.

Further features of the present disclosure will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of a configuration of an inspection systemaccording to a first exemplary embodiment.

FIG. 2 illustrates an example of a configuration of a probe according tothe first exemplary embodiment.

FIG. 3 is a block diagram illustrating an example of the configurationof the inspection system according to the first exemplary embodiment.

FIG. 4 is a flowchart illustrating overall processing of the inspectionsystem according to the first exemplary embodiment.

FIG. 5 is a flowchart illustrating measurement processing according tothe first exemplary embodiment.

FIG. 6 is a flowchart illustrating ultrasonic image processing accordingto the first exemplary embodiment.

FIG. 7 is a flowchart illustrating the measurement processing accordingto a first modification of the first exemplary embodiment.

FIG. 8 is a flowchart illustrating processing of estimating the positionand orientation of a human body according to the first exemplaryembodiment.

FIG. 9 is a flowchart illustrating processing for inspection regionidentification A according to the first exemplary embodiment.

FIG. 10 is a flowchart illustrating post-measurement processingaccording to the first exemplary embodiment.

FIG. 11 is a flowchart illustrating the measurement processing accordingto a second modification of the first exemplary embodiment.

FIG. 12 is a flowchart illustrating processing for inspection regionidentification B according to the first exemplary embodiment.

FIG. 13 is another flowchart illustrating processing for inspectionregion identification B according to the first exemplary embodiment.

FIG. 14 illustrates an image of measurement using the inspection systemaccording to the first exemplary embodiment.

FIGS. 15A, 15B, 15C, and 15D illustrate images of output resultsobtained in measurement using the inspection system according to thefirst exemplary embodiment.

FIG. 16 illustrates images of an outer appearance image and markerssupplied to the probe captured in the outer appearance image accordingto the first exemplary embodiment.

FIG. 17 illustrates an image in which skeletal information as a resultof estimating the position and orientation of the human body and a crossline as a result of estimating the position and orientation of the probeare superimposed according to the first exemplary embodiment.

FIG. 18 illustrates an image of a screen displayed on a display inultrasonic image measurement according to the first exemplaryembodiment.

FIGS. 19A, 19B, and 19C illustrate images of screens displayed on thedisplay in identifying an inspection region after ultrasonic imagedetermination according to the first exemplary embodiment.

FIG. 20 illustrates an image of an inspection state according to a thirdmodification of the first exemplary embodiment.

FIG. 21 illustrates an image of a display output result obtained inmeasurement using an inspection system according to the thirdmodification of the first exemplary embodiment.

DESCRIPTION OF THE EMBODIMENTS

[First Exemplary Embodiment]

A first exemplary embodiment will be described below with reference tothe accompanying drawings.

FIG. 1 is an overall view illustrating an ultrasonic diagnosticapparatus 100 as an example of an inspection system according to thefirst exemplary embodiment. An information processing apparatusaccording to the present disclosure is also applicable to an optionalelectronic apparatus capable of processing a captured image. Theseelectronic apparatuses may include portable phones, tablet terminals,personal computers, and information terminals of clock type or eyeglasstype.

The ultrasonic diagnostic apparatus 100 includes an ultrasonicdiagnostic apparatus body 1, an ultrasonic probe 2, a camera 3, an arm4, a display 5, and a control panel 6. The housing of the ultrasonicdiagnostic apparatus body 1 includes therein a computer includingvarious control units, a power source, and a communication interface(I/F), as an information processing apparatus.

The ultrasonic probe 2 transmits and receives ultrasonic waves in astate where an end surface thereof is in contact with the surface of asubject. The ultrasonic probe 2 includes a plurality of piezoelectricvibrators that are one-dimensionally arranged (arranged in single row)at the end surface. The ultrasonic probe 2 scans a scan area whiletransmitting ultrasonic waves into the subject body by using eachpiezoelectric vibrator, and receives reflected waves from the subject asecho signals. Usable scanning methods include B-mode scan, Doppler modescan, and any other scanning methods.

FIG. 2 illustrates an outer appearance of the ultrasonic probe 2. Theultrasonic probe 2 includes an ultrasonic probe body 201, a connector202, a marker attachment 203, a marker 204, a freeze button 6 a, and afinalization button 6 b.

A camera 3 installed at an end of the arm 4 disposed on the ultrasonicdiagnostic apparatus body 1 can be used to capture the state around theultrasonic diagnostic apparatus 100. According to the present exemplaryembodiment, the camera 3 is mainly used to capture an outer appearanceimage for identifying an inspection region at the time of inspection onthe subject using the ultrasonic probe 2. More specifically, at the timeof inspection on the subject using the ultrasonic probe 2, the camera 3captures an outer appearance image including the subject's inspectionregion and the ultrasonic probe 2.

The camera 3 includes an imaging optical system, an image sensor, acentral processing unit (CPU), an image processing circuit, a read onlymemory (ROM), a random access memory (RAM), and at least onecommunication I/F as a configuration of an ordinary camera. An image iscaptured when a light flux from the subject is formed on the imagesensor including a charge coupled device (CCD) sensor and acomplementary metal oxide (CMO) step S sensor through the imagingoptical system including lenses and other optical elements. The imagingoptical system includes a lens group. The camera 3 includes a lens drivecontrol unit for controlling zoom and focus by driving the lens groupalong the optical axis. An electrical signal output from the imagesensor is converted into digital image data by an analog-to-digital(A/D) converter. The digital image data is subjected to various imageprocessing by an image processing circuit, and the resultant is outputto an external apparatus. Instead of being subjected to the imageprocessing by the image processing circuit, the digital image data maybe output to the external apparatus via the communication I/F and thenprocessed by the processing unit of the external apparatus.

According to the present exemplary embodiment, the camera 3 uses animage sensor that mainly receives light of the visible light region tocapture an image. However, an example of the camera 3 is not limitedthereto but may be a camera that receives light of the infrared lightregion to capture an image, or a camera that receives light of aplurality of wavelength regions, such as visible light and infraredlight, to capture an image. In addition, the camera 3 may be a stereocamera capable of not only outer appearance image capturing but alsodistance measurement, or a camera having a Time-of-Flight (ToF) sensorfor distance measurement. Hereinafter, an image captured by the camera 3is referred to as a camera image.

The arm 4 is installed in the ultrasonic diagnostic apparatus body 1,and is used to dispose the camera 3 at a position and orientation at andin which an outer appearance image including the subject's inspectionregion and the ultrasonic probe 2 are enabled to be captured. Accordingto the present exemplary embodiment, the arm 4 is a serial linkmechanism arm having five joints. The joint at the end of the arm 4 thatis connected to the camera 3 is a ball joint that enables easily settingthe orientation of the camera 3.

The display 5 including a display device, such as a liquid crystaldisplay (LCD), displays images input from the ultrasonic diagnosticapparatus body 1, menu screens, and graphical user interfaces (GUIs).The display 5 displays an image stored in a memory 8 and an imagerecorded in a nonvolatile memory on the display device. The display 5 isan apparatus for displaying ultrasonic images, camera images, body markimages, probe mark images, and region identification results under thecontrol of the CPU 7. A body mark image is an image simply expressingthe body shape and is generally used for ultrasonic diagnosticapparatuses. A probe mark is displayed while being superimposed on abody mark image, and is applied for the purpose of identifying the angleat which the ultrasonic probe 2 contacts the contact surface on thesubject body.

The control panel 6 includes a keyboard, a track ball, switches, dials,and a touch panel. The control panel 6 receives various input operationsperformed by an inspector by using these operating members. Variousinput operations include an imaging instruction to perform imagecapturing using the ultrasonic probe 2 and image capturing using thecamera 3, an instruction to display various images, and an instructionto perform image switching, mode selection, and various settings. Areceived input operation signal is input to the ultrasonic diagnosticapparatus body 1 and then reflected to the control performed on eachunit by the CPU 7. If a touch panel is employed, the touch panel may beintegrated with the display 5. The inspector can make various settingsand perform operations on the ultrasonic diagnostic apparatus body 1 byperforming touch and drag operations on the buttons displayed on thedisplay 5.

When the inspector operates the freeze button 6 a in a state wheresignals are being sequentially acquired from the ultrasonic probe 2 andthe ultrasonic image in the memory 8 is being updated, signals from theultrasonic probe 2 stop, and the update of the ultrasonic image in thememory 8 is suspended. At the same time, signals from the camera 3 alsostop, and the update of the camera image in the memory 8 is suspended.In a case where the inspector operates the freeze button 6 a in a statewhere the update is stopped, signals are received from the ultrasonicprobe 2 again, the update of the ultrasonic image in the memory 8 isstarted, and the update of the camera image is also started in a similarmanner. When the inspector operates the finalization button 6 b in astate where one ultrasonic image is fixed by the depression of thefreeze button 6 a, the CPU 7 stores the ultrasonic image in thenonvolatile memory 9. The freeze button 6 a and the finalization button6 b may be provided not on the control panel 6 but on the ultrasonicprobe 2.

FIG. 3 is a block diagram illustrating a configuration of the ultrasonicdiagnostic apparatus body 1. The ultrasonic diagnostic apparatus body 1includes a transmission and reception unit 12, a signal processing unit13, an image generation unit 14, a camera control unit 15, the CPU 7,the memory 8, the nonvolatile memory 9, and a communication I/F 10 whichare all connected to an internal bus 17. These units connected to theinternal bus 17 are configured to exchange data with each other via theinternal bus 17.

The memory 8 includes a RAM (such as a volatile memory usingsemiconductor elements). The CPU 7 controls each unit of the ultrasonicdiagnostic apparatus body 1 by using the memory 8 as a work memory inaccordance with a program stored, for example, in the nonvolatile memory9. The nonvolatile memory 9 stores image data, subject data, and variousprograms for operating the CPU 7. The nonvolatile memory 9 includes ahard disk (HD) and a ROM.

The transmission and reception unit 12 includes at least onecommunication I/F for supplying power to the ultrasonic probe 2,transmitting control signals to the ultrasonic probe 2, and receiving anecho signal from the ultrasonic probe 2. The transmission and receptionunit 12 supplies to the ultrasonic probe 2 a signal for instructing theultrasonic probe 2 to emit an ultrasonic beam based on, for example, acontrol signal from the CPU 7. The transmission and reception unit 12further receives a reflection signal, i.e., an echo signal, from theultrasonic probe 2, subjects the received signal to phasing addition,and outputs the signal acquired through the phasing addition to thesignal processing unit 13.

The signal processing unit 13 includes a B-mode processing unit (orBc-mode processing unit), a Doppler mode processing unit, and a colorDoppler mode processing unit. The B-mode processing unit performsimaging on amplitude information of the receive signal supplied from thetransmission and reception unit 12 through known processing to generateB-mode signal data. The Doppler mode processing unit extracts Dopplerdeviation frequency components from the receive signal supplied from thetransmission and reception unit 12 through known processing. Then, theDoppler mode processing unit subjects the receive signal to Fast FourierTransform (FFT) processing to generate Doppler signal data as blood flowinformation. The color Doppler mode processing unit performs imaging onthe blood flow information based on the receive signal supplied from thetransmission and reception unit 12 through known processing to generatecolor Doppler mode signal data. The signal processing unit 13 outputsthe generated various data to the image generation unit 14.

The image generation unit 14 generates two- and/or three-dimensionalultrasonic images regarding the scan area through known processing,based on the data supplied from the signal processing unit 13. Forexample, the image generation unit 14 generates volume data regardingthe scan area based on the supplied data. On the basis of the generatedvolume data, the image generation unit 14 generates two-dimensionalultrasonic image data through the Multi-Planar Reconstruction (MPR)processing, and/or three-dimensional ultrasonic image data through thevolume rendering processing. The image generation unit 14 outputs thegenerated two- and three-dimensional ultrasonic images to the display 5.Examples of ultrasonic images include a B-mode image, a Doppler modeimage, a color Doppler mode image, and an M-mode image.

Each of the transmission and reception unit 12, the signal processingunit 13, the image generation unit 14, and the camera control unit 15may be implemented by hardware such as an Application SpecificIntegrated Circuit (ASIC) and a Programmable Logic Array (PLA). Inaddition, these units may be implemented by a programmable processor,such as a CPU and a Microprocessor Unit (MPU) executing software, orimplemented by a combination of software and hardware.

The camera control unit 15 includes at least one communication I/F forsupplying power to the camera 3 and transmitting and receiving controlsignals and image signals to/from the camera 3. The camera 3 may beprovided with a power source for single unit operation without beingsupplied with power from the ultrasonic diagnostic apparatus body 1. Thecamera control unit 15 can control various imaging parameters of thecamera 3, for example, zoom, focus, and diaphragm values, bytransmitting control signals to the camera 3 via the communication I/F.The camera 3 may be provided with a tripod head that can beautomatically panned and tilted, and configured to receive a pan tiltcontrol signal and perform position and orientation control through pantilt drive. In addition, a drive unit and a drive control unit forelectrically controlling the position and orientation of the camera 3may be disposed at the end of the arm 4. In such a case, the positionand orientation of the camera 3 may be controlled based on controlsignals from the camera control unit 15 or the CPU 7.

<Processing Flow>

FIG. 4 is a flowchart illustrating a processing flow performed by theCPU 7 to implement overall operations of inspection processing which isperformed by the ultrasonic diagnostic apparatus 100. More specifically,the operations in the following steps are executed by the CPU 7 or eachunit under the instruction of the CPU 7.

In step S401, the CPU 7 turns power ON in response to an inspector'soperation, and loads an operating system (OS) stored in the nonvolatilememory 9. In step S402, the CPU 7 automatically activates an ultrasonicdiagnostic application. At this timing, the CPU 7 transmits an imagesignal of the activation screen to the display 5 to display theactivation screen.

After the ultrasonic diagnostic application is activated and then theinitialization processing is completed, the CPU 7 changes the displayscreen on the display 5 to a subject information registration screen. Instep S403, the CPU 7 receives an instruction to register the subjectinformation based on an inspector's operation on the control panel 6.The subject information includes inspection region information inrelation to the condition of the subject (mammary gland, heart, artery,abdomen, carotid, thyroid gland, vein, etc.), subject identifier (ID),name, gender, date of birth, age, height, weight, and hospitalization oroutpatient. After input of the subject information, when the inspectorpresses the start button of the control panel 6 (on the display oroperation panel), the CPU 7 stores the subject information in the memory8 or the nonvolatile memory 9. Then, the CPU 7 changes the displayscreen on the display 5 to the measurement screen of the ultrasonicdiagnostic application.

In step S403, the CPU 7 also receives the setting about whether tomanually or automatically set the inspection region. The flowchart in acase where the setting for manually setting the inspection region ismade will be described below with reference to FIG. 5. The flowchart ina case where the setting for automatically setting the inspection regionis made (first and second modifications) will be described below withreference to FIGS. 7 and 11.

After the measurement screen of the ultrasonic diagnostic application isdisplayed, then in step S404, the CPU 7 performs measurement processingfor ultrasonic diagnosis in accordance with an inspector's operation.The measurement processing will be described in detail below.

Upon completion of inspection of all regions, then in step S405, the CPU7 stores inspection data obtained in the inspection in the nonvolatilememory 9 or an external medium (not illustrated) or transfers theinspection data to an external apparatus (external server) via thecommunication I/F 10.

When the entire processing is completed and the inspector turns powerOFF, then in step S406, the CPU 7 performs processing of ending theultrasonic diagnostic application and the OS to complete the series ofprocessing.

FIG. 5 is a flowchart illustrating a processing flow of the measurementprocessing in step S404 illustrated in FIG. 4 in a case where theinspector makes the setting for manually setting the inspection region.Each of the following steps is executed by the CPU 7 or each unit underthe instruction of the CPU 7.

In step S501, the CPU 7 subjects the echo signal received from theultrasonic probe 2 to signal processing and image processing to generatean ultrasonic image, and displays the ultrasonic image on the display 5.The ultrasonic image processing in step S501 will be described in detailbelow.

When the inspector checks the ultrasonic image displayed on the display5, and if a desired ultrasonic image becomes obtainable, the CPU 7finalizes the ultrasonic image based on an inspector's operation. Instep S502, the CPU 7 stores the ultrasonic image in the memory 8.

In step S503, to record information about where the inspection regionis, inspection regions such as body marks and probe marks are set inaccordance with an inspector's operation. Annotations, such as commentsand arrows, may be input to the display 5 in accordance with aninspector's operation.

When the inspector presses the finalization button 6 b, then in stepS504, the CPU stores the information about the inspection region in thememory 8.

Upon completion of the measurement processing for a certain inspectionregion, then in step S505, the CPU 7 determines whether the measurementis completed for all the inspection regions predetermined based oninspection items. If the inspection is not completed for any inspectionregion (NO in step S505), the processing returns to step S501. Then, theCPU 7 performs the measurement processing again. The information aboutinspection items is classified and recorded in the nonvolatile memory 9based on the inspection region and symptom in advance. Then, theinformation is selected from classified and recorded information and setbased on an inspector's operation. If the measurement processing iscompleted for all the inspection regions (YES in step S505), the CPU 7ends the processing in step S404.

FIG. 6 is a flowchart illustrating a detailed flow of the ultrasonicimage processing in step S501. Each of the following steps is executedby the CPU 7, or the signal processing unit 13 and the image generationunit 14 under the instruction of the CPU 7.

As described above, the ultrasonic probe 2, which is an example of aninspection device, scans the scan area while transmitting ultrasonicwaves into the subject body by using each piezoelectric vibrator, andreceives the reflected wave from the subject as an echo signal(ultrasonic signal). According to the present exemplary embodiment, theultrasonic probe 2 is configured to be operable by the inspector withthe ultrasonic probe 2 in their hand. In step S601, the signalprocessing unit 13 and the image generation unit 14 subject theultrasonic signals transmitted from the ultrasonic probe 2, to signalprocessing and image processing, respectively, to generate an ultrasonicimage. The CPU 7 then displays the ultrasonic image on the display 5. Toobtain the desired image, the inspector can further adjust and correctvarious processing parameters by using the control panel 6 whilemonitoring the ultrasonic image displayed on the display 5. Morespecifically, in step S602, various parameters (mode, gain, focus, echolevel, etc.) are changed based on operation signals accepted by thecontrol panel 6, and the ultrasonic image after the change isregenerated and displayed on the display 5.

In step S603, the CPU 7 determines whether the freeze button 6 aprovided on the ultrasonic probe 2 is pressed. If the freeze button 6 ais not pressed (NO in step S603), the CPU 7 repeats steps S601 and S602.If the freeze button 6 a is pressed (YES in step S603), the CPU 7determines that the desired ultrasonic image has been acquired, displaysthe ultrasonic image captured and generated at this timing on thedisplay 5. The CPU 7 then ends the ultrasonic image processing.

FIG. 7 is a flowchart illustrating another form of the detailed flow ofthe measurement processing in step S404 illustrated in FIG. 4 in a casewhere the inspector makes the setting for automatically setting theinspection region. Each of the following steps is executed by the CPU 7or each unit under the instruction of the CPU 7.

According to a first modification, in step S503 illustrated in FIG. 5,the CPU 7 automatically sets the inspection region performed based on aninspector's operation.

In step S701, the CPU 7 estimates the position and orientation of thesubject lying on a bed based on the image acquired by the camera 3. Whena result of estimating the position and orientation of the subject isfinalized based on an inspector's operation, the CPU 7 stores theposition and orientation of the subject in the memory 8. This processingwill be described in detail below. In step S702, the CPU 7 estimates theposition and orientation of the probe 2 by using the image acquired bythe camera 3, automatically identifies the subject's inspection regionbased on the estimation result for the subject in step S701, anddisplays the inspection region on the display 5. This processing will bedescribed in detail below.

In step 703, when the freeze button 6 a on the control panel 6 or theultrasonic probe 2 is pressed by the inspector, the update of theultrasonic image on the display 5 is stopped. If the ultrasonic image isthe desired one, the processing proceeds to step S704 based on thedepression of the finalization button 6 b by the inspector. In stepS704, the CPU 7 performs post-measurement processing. Upon completion ofthe measurement processing for a certain inspection region, then in stepS705, the CPU 7 determines whether the measurement is completed for allthe inspection regions predetermined based on the inspection items. Ifthe inspection is not completed for any inspection region (NO in stepS705), the processing returns to step S701.

According to the present exemplary embodiment as described above, if aninspection region is identified in step S702 before an ultrasonic imagein step S1002 is captured, the inspector can concentrate on thedetermination of the ultrasonic image when capturing an ultrasonic imagein step S1002. Although the CPU 7 acquires a camera image in step S703,this camera image is used only for the estimation of the position andorientation of the probe 2 and the screen display but not used for theinspection region identification. The processing in step S703 will bedescribed in detail below with reference to FIG. 10.

FIG. 8 is a flowchart illustrating a detailed flow of processing ofestimating the position and orientation of the subject body in step S701illustrated in FIG. 7. Each operation in the following steps is executedby the CPU 7 or each unit under the instruction of the CPU 7.

In step S801, the CPU 7 instructs the camera control unit 15 to capturethe subject lying on the bed by using the camera 3, as illustrated inFIG. 15A. The camera 3 sequentially performs imaging at a predeterminedframe rate, and the CPU 7 receives outer appearance images via thecommunication I/F of the camera control unit 15, and sequentiallydisplays the images on the display 5.

The inspector adjusts the position of the arm 4 so that at least aninspection region of interest, which is a part of the subject, fits intothe angle of field of the camera 3, while checking the outer appearanceimage displayed on the display 5. The display 5 may display a line forguiding the inspector where to bring the position of the subject'sinspection region in the displayed outer appearance image by adjustingthe position of the camera 3. In such a case, the line for guidance(i.e., GUI data to be superimposed on the display image) is prestored inthe nonvolatile memory 9 in association with the information about theinspection region.

In step S802, the CPU 7 estimates the position and orientation of thesubject body (human body) through the image analysis processing based onthe image from the camera 3 acquired as an outer appearance image.According to the present exemplary embodiment, the CPU 7 outputsskeletal information including the position coordinates of a featurepoint of each joint as the position and orientation information for thesubject body (human body). Examples of joints include the nose, neck,right shoulder, right elbow, right wrist, left shoulder, left elbow,left wrist, central buttock, right buttock, right knee, right ankle,left buttock, left knee, left ankle, right eye, left eye, right ear,left ear, left thumb, left little finger, left heel, right thumb, rightlittle finger, and right heel. The CPU 7 uses a learner trained througha machine learning (deep learning) method as means for obtaining theskeletal information based on an image. According to the presentexemplary embodiment, the CPU 7 uses a learning model (learner) trainedwith a set of a plurality of training images including a human body as asubject and correct answer information for skeletal information(probability distribution of joints) with respect to each trainingimage. This method makes it possible to obtain the skeletal informationin two-dimensional (2D) or three-dimensional (3D) coordinates regardlessof whether information obtained from the camera 3 (including a stereocamera, infrared camera, and TI/F camera) is only a luminance image,only a depth image, or both images. Examples of known learners of thistype include Openpose (registered trademark) of Carnegie MellonUniversity. According to the present exemplary embodiment, position andorientation information (skeletal information) estimated by the learnertrained through machine learning is stored in a memory in array or listform. More specifically, for a plurality of regions such as theabove-described joints, information indicating the distribution of theprobability that each region exists in an image is output as estimatedposition and orientation information. When the distribution (probabilitydistribution) of the reliability that a region n (n is an integer)exists in an image is Rn(x, y), the output R as skeletal information isrepresented by R={Rn(x, y)|n=1, 2, . . . , N (N is an integer)}. Rn(x,y) does not need to be the reliability distribution for the entire areaon the image but may be the reliability distribution only for an areahaving a reliability larger than a threshold value. Alternatively, Rn(x,y) may be stored only a peak value of the reliability in associationwith the coordinates (e.g., region 3: right shoulder, reliability: 0.5,coordinates: (x, y)=(122, 76)).

The CPU 7 extracts the position of the peak value of the reliability foreach region (i.e., position at which each region is detected with thehighest probability of existence) by using the output R. FIG. 15Billustrates an example of skeletal information visualized based on thisinformation.

As preprocessing of obtaining the skeletal information by using thelearner trained with images, the present exemplary embodiment performsimage quality correction such as noise rejection, distortion correction,color conversion, and luminance and color gradation correction, andperforms image rotation and flipping. Parameters for image correctionare tabulated based on, for example, the model of the camera 3 and animaging condition, and then prestored in the nonvolatile memory 9. Thecorrection processing is performed on input outer appearance image byusing these correction parameters to bring the imaging condition closeto the imaging condition of images of the data set used in training, sothat the estimation is performed with higher accuracy. For example, whenan image captured in a dark room is corrected to a brighter image,high-sensitivity noise may occur, resulting in a different tendency fromthe data set used in learning. In such a case, it is desirable thatprocessing of removing high-sensitivity noise be performed on the inputouter appearance image. Likewise, in a case where the lens of the camera3 is a wide-angle lens having a large peripheral distortion, it isdesirable that the distortion correction be performed on the input outerappearance image. In addition, in a case where all images in the dataset includes the head at the top, it is desirable that an image berotated or flipped so that the head comes to the top and the resultantbe input. If the training is performed by using images converted throughcertain processing, it is desirable that the input image be convertedbefore input to the learner.

As illustrated in FIG. 15B, skeletal information for the inspector andneighboring persons may also be acquired together in step S802. Thus, instep S803, the CPU 7 identifies the skeletal information for the subjectbased on the image of the skeletal information acquired in step S802.

Conceivable methods for identifying the skeletal information about thesubject include the following examples. One method is to combine withface recognition processing. The CPU 7 authenticates the inspector'sface pre-registered in the nonvolatile memory 9 from the outerappearance image. The skeletal information about the inspector and theskeletal information about the subject are then identified from thedistance relation between the position where the inspector's face isauthenticated and detected and the portions detected as facial regions(such as eyes, nose, and ears) in the skeletal information detected instep S802.

As another method, the CPU 7 may identify the ultrasonic diagnosticapparatus body 1 and distinguish between the inspector and the subjectbased on the planar (X- and Y-directional) or three-dimensional(Z-directional) distance relation from the ultrasonic diagnosticapparatus body 1. As still another method, the CPU 7 identifies theorientation type based on the positional relation between joint pointsin the skeletal information to distinguish between the inspector and thesubject. As still another method, a procedure is set so that the subjectis image-captured in a predetermined area in adjusting the angle offield of the camera 3, and the CPU 7 identifies the subject based on theposition in the skeletal information.

According to the present exemplary embodiment, the CPU 7 performsprocessing through at least one of the above-described identificationmethods to identify the inspector and the subject and visualize theposition and orientation information for the identified subject, asillustrated in FIG. 15C.

In step S804, the CPU 7 displays on the display 5 an image in which theposition and orientation information for the subject thus estimated issuperimposed on the display image from the camera 3, as illustrated FIG.15D.

For the image to be displayed on the display 5, the CPU 7 may notdirectly use the image captured by the camera 3. Instead, inconsideration of privacy, the CPU 7 may replace the image with an avataror animation or convert the image into a 3D model through known imageprocessing.

In step S805, the CPU 7 determines whether the freeze button 6 a ispressed. If the freeze button 6 a is pressed by the inspector (YES instep S805), the CPU 7 ends the update of the estimation result for theskeletal information (position and orientation information) for thesubject based on the outer appearance images sequentially displayed onthe display 5. The processing then proceeds to step S806. If the freezebutton 6 a is not pressed (NO in step S805), the processing returns tostep S801. The CPU 7 continues the estimation of the position andorientation information.

After the freeze button 6 a is pressed (YES in step S805), then in stepS806, if the inspector makes sure that there is no problem in theestimation result for the position and orientation information displayedon the display 5 and then operates the finalization button 6 b, the CPU7 ends the processing for estimating the position and orientation of thesubject body. If the desired result of the position and orientation isnot acquired (NO in step S806), the processing returns to step S801.Then, the CPU 7 repeatedly performs processing for estimating theposition and orientation.

In step S806, the CPU 7 waits for the confirmation by the inspector inorder to address a case where the orientation of the subject is not thedesired one, a case where the camera angle is not preferable, or a casewhere the result of detecting the position and orientation is remarkablydifferent from the visually confirmed position and orientation.

In other exemplary embodiments, steps S805 and S806 may be omitted(ignored). More specifically, the position and orientation informationfor the subject body (human body) may be kept being updated atpredetermined time intervals until the inspector determines that theprocessing for identifying the inspection region in the subsequent stageis suitable and then performs the finalizing operation.

Although, in the present exemplary embodiment, the CPU 7 uses theskeletal information (joint information) as a method for estimating theposition and orientation of the subject body (human body), the CPU 7 mayestimate the 3D shape of the subject body based on mesh-formedinformation. Such a technique can be implemented by using machinelearning (a learner). More specifically, this method may use a learnerthat trained by using a set of a plurality of training images includingthe subject and correct answer information for the mesh-formedinformation for each training image.

FIG. 9 is a flowchart illustrating a detailed operation of theinspection region identification A in step S702 illustrated in FIG. 7.Each operation in the following steps is executed by the CPU 7 or eachunit under the instruction of the CPU 7.

In step S901, the CPU 7 acquires an outer appearance image (image data)including the ultrasonic probe 2 from the camera 3 via the communicationI/F of the camera control unit 15. An image may be captured withoutchange since the angle of view of the camera 3 has been adjusted toinclude the subject in the previous step. However, at least one of pancontrol, tilt control, and zoom control may be performed to achieve anangle of view at which the ultrasonic probe 2 is easier to be detected.FIG. 16 illustrates a change in the angle of view of the camera 3 in acase where the angle of view is adjusted from the outer appearance imagecentering on the subject to the angle of view centering on theultrasonic probe 2, and a captured image.

In step S902, the CPU 7 analyzes the acquired outer appearance image toobtain the position and orientation of the ultrasonic probe 2. Morespecifically, the CPU 7 detects a plurality of Argument Reality (AR)markers (a plurality of corners is provided on each marker) provided onthe ultrasonic probe 2 through image recognition processing includingfilter processing, binary processing, determination processing, andshape recognition processing for edge detection based on the outerappearance image. The CPU 7 may subject the outer appearance imageacquired by the camera 3 to sharpness adjustment, gain adjustment, noisereduction, and other image processing to perform image recognition withhigher accuracy. The CPU 7 detects the AR markers provided on theultrasonic probe 2 and calculates the position and orientation of theultrasonic probe 2 based on the positional relation of a plurality ofcorners existing on the detected AR markers, sizes of drawings formed bythe corners, and distortion conditions. Since the ultrasonic probe 2 isrigid, the relation between each AR marker and the inspection surface ofthe ultrasonic probe 2 can be obtained through calculation. Asillustrated in FIGS. 16 and 19B, it is desirable that a plurality of ARmarkers (desirably at least three AR markers) is disposed on the markerattachment 203. The AR markers are disposed at predetermined intervalson the outer periphery of the ultrasonic probe 2 so that at least one ofthe AR markers captured by the camera 3 can be acquired regardless ofthe orientation of the ultrasonic probe 2 and the position of theconnector 202. According to the present exemplary embodiment, the CPU 7outputs the following data as an output regarding the position andorientation of the ultrasonic probe 2. The CPU 7 outputs position andorientation information including positional information (imagecoordinate information (x, y)) in the outer appearance image, positionalinformation (x, y, z) in the three-dimensional real space with referenceto the camera 3, and vector information (direction vector d=(x, y, z))representing the orientation of the probe 2. The positional information(x, y) is used for processing for identifying the inspection region instep S903. The position and orientation information (x, y, z), d is usedfor performing display such that the position and orientation of theprobe 2 with respect to the inspection region in the body mark in thescreen displayed on the display 5 in step S904 is visually identified.However, the use of the position and orientation information is notlimited thereto. The CPU 7 may use the position and orientationinformation in the processing for identifying the inspection region andmay display only the positional information in the display processing.

If the position and orientation of the ultrasonic probe 2 can beobtained through calculation, the CPU 7 may use light emitting diodes(LEDs), retroreflection marks, and any other marks, instead of ARmarkers, as criteria (indexes) for obtaining the position andorientation of the ultrasonic probe 2.

Since the ultrasonic probe body 201 generally has a rigid body and thusthe position and orientation thereof are limited, the ultrasonic probebody 201 detects the position and orientation through rule-basedprocessing, i.e., image recognition using predetermined patterns such asAR markers. This makes it possible to easily achieve high detectionaccuracy and obtain the position and orientation with lower processingcost and higher processing speed than those of the technique using alearner trained through machine learning.

In step S903, the CPU 7 identifies the inspection region based on therelation between the estimation result (R) for the position andorientation of the subject obtained in step S701 and stored in thememory 8 and the positional information (x, y) for the ultrasonic probe2 obtained in step S902. According to the present exemplary embodiment,the CPU 7 outputs a plurality of inspection region candidates as aresult of identification, starting from the inspection region having thehighest evaluation value for the inspection region in each piece ofskeletal information.

The following methods are applicable for a method for identifying theinspection region in a case where skeletal information R has beenacquired as position and orientation information for the subject.

For example, the CPU 7 extracts the position having a peak value of thereliability in the reliability distribution Rn(x, y) for each region,i.e., the coordinates of the position having the highest reliability foreach region (simply referred to as the coordinates of each region). TheCPU 7 then identifies the inspection region based on the coordinates ofeach region and the distance from the positional information(Euclid/Mahalanobis distance) about the ultrasonic probe 2. Morespecifically, the CPU 7 calculates the evaluation value so that theevaluation value increases with decreasing distance between thecoordinates of each region and the ultrasonic probe 2. The CPU 7 alsocalculates the evaluation value for each region so that a greater weightis applied as a reliability corresponding to the coordinates of eachregion increases. The CPU 7 then sequentially extracts inspection regioncandidates from among a plurality of regions, starting from the regionhaving the relatively highest evaluation value. In the evaluation valuecalculation, the CPU 7 may refer to only either one of the distance fromthe position of the ultrasonic probe 2 and the reliability distributionfor each region.

The CPU 7 may calculate the evaluation value in each area in the imageby using the reliability distribution Rn(x, y) for each region. Morespecifically, the CPU 7 may calculate the evaluation value distributiondetermined with Rn(x, y)*(Weight based on the distance from theultrasonic probe 2) for each region, and then identify the positionhaving the highest evaluation value and the region at that position asan inspection region.

Examples of other methods for identifying the inspection region includecalculating the evaluation value based on the distance between astraight line connecting the positions of the above-described regionsand the positional information (x, y) about the ultrasonic probe 2, andthe reliability distribution of the two regions corresponding to thestraight line.

Examples of still other methods for identifying the inspection regioninclude calculating the evaluation value based on the coordinatesobtained by dividing the positions of a plurality of regions by acertain ratio, the distance from the positional information (x, y) forthe ultrasonic probe 2, and the reliability distribution of the regionsat two points of the division source.

Examples of still other methods for identifying the inspection regionalso include generating a closed 2D or 3D domain by using the pointsobtained with a certain ratio based on the relation between a pluralityof joints, and then calculating the evaluation value by subjecting theclosed domain to inside/outside determination on the positionalinformation (x, y) about the ultrasonic probe 2.

The CPU 7 may calculate the evaluation value by combining a part or allof the plurality of the above-described methods.

In step 903, the position and orientation of the subject stored in thememory 8 may be different from the current position and orientation ofthe subject because the subject has moved after execution of step 701.Inspection regions identified by using different position andorientation may be an incorrect result. Thus, the CPU 7 may determinewhether the subject is moving, and, if a movement is detected, theprocessing returns to step 701. The CPU 7 may estimate the position andorientation of the subject body again.

Examples of methods for determining whether the subject is movinginclude using a differential image or using an optical flow.

In a case where a differential image is used, for example, the CPU 7subjects a portion of the inspector's hand and the probe 2 to maskprocessing. Then, in the images acquired in steps S801 and S901, the CPU7 determines whether the remaining portion has changed in the luminancevalue and hue by an amount equal to or larger than threshold values. Insuch a case, the CPU 7 may detect the above-described changes throughstatistical processing.

In a case where an optical flow is used, for example, the CPU 7registers as a pattern the subject body (human body) in the camera imagethat is acquired in step S801 when the position and orientation of thesubject body is estimated. The CPU 7 then subjects the camera image instep S901 to template matching to detect the movement. Alternatively,the CPU 7 temporarily stores the camera image acquired in step S801 inthe memory 8. Then, the CPU 7 calculates the amount of movement of afeature point determined through SHIFT and AKAZE in the image acquiredin step S901 and between the two images, thus detecting the movement ofthe human body in an image. In addition, known tracking methods such asKCF tracker is also applicable.

In step S904, the CPU 7 displays the inspection region identified instep S903 on the display 5. In step S905, the CPU 7 determines whetherthe inspector performs an operation corresponding to OK on the controlpanel 6 or presses the freeze button 6 a on the ultrasonic probe 2.

FIG. 21 illustrates an example of a screen displayed on the display 5when the inspector acknowledges the result of the inspection regionidentification in step S905. The CPU 7 displays the superposition of abody mark 1901 of the GUI corresponding to the subject body and a probemark 2102 of the GUI corresponding to the ultrasonic probe 2 at theposition of the corresponding inspection region. In such a case, boththe body mark and the probe mark may be displayed if the name of theinspection region has been identified (“Septal Leaflet” in FIG. 21). Thescreen displays an inspection result confirmation window including an OKand a Redo icon button. The inspector selects the OK or the Redo buttonby using the control panel 6 or presses the freeze button 6 a on theultrasonic probe 2 to finalize the inspection region or issue a redoinstruction. If the inspector selects the OK button or presses thefreeze button 6 a (YES in step S905), the CPU 7 ends the series ofprocessing. If the inspector selects the Redo button or does not pressesthe freeze button 6 a (NO in step S905), the processing returns to stepS901. The CPU 7 repeats the processing for identifying the inspectionregion.

While the window for promoting the inspection result confirmationillustrated in FIG. 21 in step S905 is being displayed, the processingfor estimating the inspection region may be suspended. The finalizationprocessing in step S905 may be omitted, and the processing of finalizingthe inspection region may be performed at an optional timing in stepsS701 to S704.

In step S906, the CPU 7 finalizes the inspection region identified instep S903 in response to an inspector's operation. The inspectorconfirms the inspection region displayed on the display 5 and, if thedesired inspection region is obtained, operates a predeterminedoperating member of the control panel 6 or presses the finalizationbutton 6 b on the ultrasonic probe 2. If the identified inspectionregion is wrong, the inspector operates the control panel 6 to displaythe second and third region candidates on the display 5. Then, when theinspector selects an inspection region candidate, the selectedinspection region candidate is finalized.

FIG. 10 is a flowchart illustrating in detail a flow of the ultrasonicimage capturing in step S703 illustrated in FIG. 7.

FIG. 10 illustrates processing that combines the ultrasonic imageprocessing illustrated in FIG. 6 and the processing of inspection regionidentification A illustrated in FIG. 9. For steps performing basicallythe same operations as the processing of the flowcharts illustrated inFIGS. 6 and 9, the descriptions will be omitted. Step S1001 correspondsto step S601, step S1002 corresponds to step S602, step S1003corresponds to step S901, step S1004 corresponds to step S902, stepS1005 corresponds to step S904, step S1006 corresponds to step S603, andstep 1007 corresponds to step S604.

FIG. 11 is a flowchart illustrating in detail a flow of thepost-measurement processing in step S704 illustrated in FIG. 7. Each ofthe following steps is executed by the CPU 7 or each unit under theinstruction of the CPU 7.

In step S1101, the CPU 7 stores the ultrasonic image finalized in stepS1007 in the nonvolatile memory 9 or an external medium, or transmitsdata to the outside.

In step S1102, the CPU 7 stores the information about the inspectionregion finalized in step S906 and the position and orientationinformation (collectively referred to as inspection region information)about the probe 2 in the nonvolatile memory 9 or an external medium, ortransmits data to the outside in association with the ultrasonic imagein step S1101. Examples of information about the inspection region andinformation about the position and orientation of the probe 2 includethe name of the inspection region, the body mark, and the position andorientation information for the probe mark with respect to the bodymark. The position and orientation information for the probe mark may bestopped at the angle on the two-dimensional image or may bethree-dimensional orientation information. The inspection regioninformation does not necessarily need to include the position andorientation information for the probe 2.

(Second Modification)

FIG. 12 is a flowchart illustrating another form of step S404illustrated in FIG. 4 in a case where the inspector makes the settingfor automatically setting the inspection region. Step S1203 correspondsto step S703, step S1204 corresponds to step S704, and step S1205corresponds to step S705. For steps performing basically the sameoperations as the processing of the flowchart illustrated in FIG. 7,descriptions thereof will be omitted.

In this flowchart, as in the flowchart illustrated in FIG. 7, the CPU 7automatically sets the inspection region performed based on aninspector's operation in step S503 illustrated in FIG. 5.

FIG. 13 is a flowchart illustrating a detailed flow of inspection regionidentification B in step S1202 illustrated in FIG. 12. For stepsperforming basically the same operations as the processing of theflowcharts illustrated in FIGS. 8 and 9, descriptions will be omitted.

FIG. 13 illustrates processing that combines the processing forestimating the position and orientation of the subject body illustratedin FIG. 8 and the inspection region identification A illustrated in FIG.9. Step S1301 corresponds to step S901, step S1302 corresponds to stepS802, step S1303 corresponds to step S803, step S1304 corresponds tostep S902, step S1305 corresponds to step S903, step S1306 correspondsto S904, step S1307 corresponds to step S905, and step S1308 correspondsto step S906. The flowchart illustrated in FIG. 13 differs from theflowchart illustrated in FIG. 9 in that steps S802 and S803 illustratedin FIG. 8, which correspond to processing for estimating the positionand orientation of the subject body, are added. Referring to theflowchart illustrated in FIG. 7, this processing is performed outsidethe flowchart.

In the processing illustrated in FIG. 13, the CPU 7 performs processingfor estimating the position and orientation of the subject body andestimating the position and orientation of the probe 2 from a singleouter appearance image acquired in a single image capturing, unlike theprocessing illustrated in FIG. 7, to estimate the inspection region.This enables reducing the number of inspector's operations for pressingthe freeze button 6 a and pressing the finalization button 6 b. However,there arises an issue that, in a case where subject's inspection regionsand the ultrasonic probe 2 cannot be image-captured at the same time,the inspection region identification becomes difficult. For example, theinspection region may be hidden by the ultrasonic probe 2 or theinspector. By contrast, in the processing illustrated in FIG. 7, thecamera image for estimating the position and orientation of the subjectbody is different from the camera image for estimating the position andorientation of the probe 2, making it comparatively easier to avoid theissue.

(Third Modification)

A modification of the processing for identifying the inspection regionperformed in step S703 illustrated in FIG. 7, step S1102 illustrated inFIG. 11, and steps S1302 and S1304 illustrated in FIG. 13 will bedescribed below. According to the above-described exemplary embodiment,the CPU 7 performs each operation assuming the inspection regionidentification for the entire human body. However, this also applies toa local inspection target, such as a hand.

For example, ultrasonographic inspection targeting a symptom ofrheumatism is performed on joints of hands and feet of the subject.Thus, the learner trained for estimation of the position and orientationof a human body according to the present modification targets alocalized inspection region. Generally, a learner for outputtinginformation about the skeletal frame (joints) of hands and feet from aninput image is used, which is different from a learner targeting theentire human body. Alternatively, a learner capable of detecting theskeletal frame of the human body, hands, and feet at the same time maybe used.

As described above, since a plurality of candidates of localizedinspection targets, e.g., hands and feet, is presumed, there is prepareda plurality of learners each corresponding to respective regions todetect the skeletal information for each region. In performingprocessing for identifying the inspection region, it is desirable thatswitching of the plurality of prepared learners is automaticallyperformed based on and the outer appearance image acquired by the camera3, or that information about the target region be selected and input inadvance through an inspector's operation and switching of the pluralityof prepared learners is performed.

A hand has five fingers (thumb, index finger, middle finger, fourthfinger, and little finger) each of which has the first to the fourthjoints. The trained learner according to the present modificationestimates the position for each joint.

The output of the skeletal information may include finger-tips inaddition to joints. The fourth joint is set as the same single point ina representative basis for all of the five fingers.

FIG. 18 illustrates an image of an inspection state when the inspectiontarget region is any one region of the hand. The camera 3 is disposedand the position and orientation of the camera 3 are controlled so thatthe subject's inspection region and the ultrasonic probe 2 fit into theangle of view.

FIG. 19A illustrates an example of an image displayed on the display 5in a hand inspection according to the present modification. The image isobtained by the CPU 7 estimating the skeletal information about a handas the processing for estimating the position and orientation of thesubject body, and superimposing the skeletal information on the outerappearance image obtained from the camera 3.

FIG. 19B illustrates an example of an image displayed on the display 5in the hand inspection state according to the present modification. Theimage is obtained by the CPU 7 superimposing a result (cross line) ofestimating the position and orientation of the probe 2 on the imageacquired by the camera 3.

FIG. 19C illustrates an example of an image in which a result ofestimating the position and orientation of the subject body and a resultof estimating the position and orientation of the probe 2 are displayedtogether on the display 5. Through processing similar to the processingin step S903, it is possible to identify the inspection region, i.e.,which joint of which finger.

As described above, according to the present exemplary embodiment, theCPU 7 estimates the detection of the position and orientation of asubject body (human body) in inspection by using a learner obtainedthrough machine learning, and identifies the inspection region based ona result of this estimation and a result of estimating the position ofan inspection device, thus achieving inspection region identificationwith higher accuracy.

According to the present exemplary embodiment, the CPU 7 performsdetection processing with a lower processing load than that for theprocessing of detecting the position and orientation of a human body ona rule basis in estimating the position of the inspection device. Thismakes it possible to offer an inspection system having a low processingload while the system tracks the position of the probe 2 having a largeamount and high frequency of movement.

The present exemplary embodiment also makes it possible to more smoothlyidentify the inspection region. The present exemplary embodiment alsoenables the inspector to perform operations while constantly graspingthe region currently being inspected, preventing a region input failure.

(Other Exemplary Embodiments)

The object of the present disclosure can also be achieved by thefollowing configuration. A storage medium storing a program code ofsoftware describing procedures for implementing the functions of theabove-described exemplary embodiments is supplied to a system orapparatus. Then, a computer (or CPU or MPU) of the system or apparatusreads the program code stored in the storage medium and then executesthe program code.

In such a case, the program code itself read from a storage mediumimplements new functions of the present disclosure, and the storagemedium storing the program code and the program are also included in thepresent disclosure.

Examples of storage media for supplying a program code include aflexible disk, hard disk, optical disk, and magneto-optical (MO) disk.In addition, a compact disc read only memory (CD-ROM), compact discrecordable (CD-R), compact disk rewritable (CD-RW), digital versatiledisc read only memory (DVD-ROM), digital versatile disc random accessmemory (DVD-RAM), digital versatile disc rewritable (DVD-RW), digitalversatile disc recordable (DVD-R), magnetic tape, nonvolatile memorycard, and ROM are also applicable.

The functions of the above-described exemplary embodiments areimplemented by the computer making the read program code executable.Further, a case where an OS operating on the computer performs a part orall of actual processing based on instructions of the program code, andthe functions of the above-described exemplary embodiments areimplemented by the processing is also included in the presentdisclosure.

The following case is also included in the present disclosure. First ofall, a program read from a storage medium is written in a memoryincluded in a function expansion board inserted into the computer or afunction expansion unit connected to the computer. Subsequently, a CPUincluded in the function expansion board or function expansion unitexecutes a part or all of actual processing based on instructions of theprogram code.

Embodiment(s) of the present disclosure can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present disclosure has been described with reference toexemplary embodiments, it is to be understood that the disclosure is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2019-154070, filed Aug. 26, 2019, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An information processing apparatus comprising:at least one processor; and a memory storing instructions that, whenexecuted by the at least one processor, cause the at least one processorto: acquire a first image including at least a part of an inspectiondevice and a second image including at least a part of a subject, thefirst and the second images being captured by an imaging apparatus;estimate, as a first estimation, a position of the inspection devicebased on the first image; estimate, as a second estimation, position andorientation information for the subject based on the second image;identify an inspection region of the subject from a result of the firstestimation and a result of the second estimation; and receive a firstfinalization instruction to finalize an inspection result to be recordedor output to an external apparatus based on sequentially acquiredinspection results of the subject, and a second finalization instructionto finalize inspection region information to be recorded or output tothe external apparatus in association with the inspection result,wherein the second finalization instruction for the inspection regioninformation is received before the first finalization instruction forthe inspection result.
 2. The information processing apparatus accordingto claim 1, wherein, before receiving the first finalization instructionfor the inspection result, the sequentially acquired inspection resultsand the inspection region information based on the received secondfinalization instruction are displayed on a display device.
 3. Theinformation processing apparatus according to claim 1, wherein an outputof the second estimation is the position and orientation information forthe subject with the second image as an input, based on a learning modeltrained by using a plurality of training images including subjectssimilar to the subject.
 4. The information processing apparatusaccording to claim 1, wherein an output of the first estimation is anestimate of the position of the inspection device through filterprocessing and shape recognition processing.
 5. The informationprocessing apparatus according to claim 1, wherein an output of thesecond estimation is skeletal information indicating a plurality ofjoint points of the subject and positions thereof as the position andorientation information for the subject.
 6. The information processingapparatus according to claim 5, wherein execution of the instructionsfurther configures the at least one processor to identify the inspectionregion of the subject based on a distance from the position of theinspection device estimated by the first estimation unit to theplurality of joint points.
 7. The information processing apparatusaccording to claim 6, wherein execution of the instructions furtherconfigures the at least one processor to calculate an evaluation valueby applying a larger weight to a joint point less distant from theposition of the inspection device estimated by the first estimation, andidentifies the inspection region of the subject based on the evaluationvalue calculated for each of the plurality of joint points.
 8. Theinformation processing apparatus according to claim 6, wherein thesecond estimation outputs information about reliability for each of theplurality of joint points of the subject, and wherein the inspectionregion of the subject is identified based on the information about thereliability.
 9. The information processing apparatus according to claim1, wherein a processing load associated with estimating the position ofthe inspection device using the first image during the first estimationis smaller than a processing load associated with estimating theposition and orientation of the subject using the second image duringthe second estimation.
 10. The information processing apparatusaccording to claim 1, wherein the first and the second images areacquired from an image acquired in a single image capturing.
 11. Theinformation processing apparatus according to claim 1, wherein theposition of the inspection device is detected during the firstestimation by detecting a marker disposed on the inspection device fromthe first image.
 12. The information processing apparatus according toclaim 1, wherein the position and orientation of the inspection deviceis detected during the first estimation by detecting a marker disposedon the inspection device from the first image.
 13. The informationprocessing apparatus according to claim 1, wherein the sequentiallyacquired inspection results are acquired via an ultrasonic probe.
 14. Aninspection system comprising: an imaging apparatus configured to capturea first image including at least a part of an inspection device and asecond image including at least a part of a subject; at least oneprocessor; and a memory storing instructions that, when executed by theat least one processor, cause the at least one processor to: inspect thesubject; estimate, as a first estimation, a position of the inspectiondevice based on the first image; estimate, as a second estimation,position and orientation information for the subject based on the secondimage; identify an inspection region of the subject from a result of thefirst estimation and a result of the second estimation; and receive afirst finalization instruction to finalize an inspection result to berecorded or output to an external apparatus based on sequentiallyacquired inspection results, and a second finalization instruction tofinalize inspection region information to be recorded or output to theexternal apparatus in association with the inspection result, whereinthe second finalization instruction for the inspection regioninformation is received before the first finalization instruction forthe inspection result.
 15. An information processing method comprising:acquiring a first image including at least a part of an inspectiondevice and a second image including at least a part of a subject, thefirst and the second images being captured by an imaging unit;estimating, as first estimating, a position of the inspection devicebased on the first image; estimating, as second estimating, position andorientation information for the subject based on the second image;identifying an inspection region of the subject from a result of thefirst estimating and a result of the second estimating; and receiving afirst finalization instruction to finalize an inspection result to berecorded or output to an external apparatus based on sequentiallyacquired inspection results of the subject, and a second finalizationinstruction to finalize inspection region information to be recorded oroutput to the external apparatus in association with the inspectionresult, wherein, the second finalization instruction for the inspectionregion information is received before the first finalization instructionfor the inspection result.
 16. A non-transitory computer-readablestorage medium storing instructions that, when executed by at least oneprocessor of an information processing apparatus, configure the at leastone processor of the information processing apparatus to performoperations comprising: acquire a first image including at least a partof an inspection device and a second image including at least a part ofa subject, the first and the second images being captured by an imagingapparatus; estimate, as a first estimation, a position of the inspectiondevice based on the first image; estimate, as a second estimation,position and orientation information for the subject based on the secondimage; identify an inspection region of the subject from a result of thefirst estimation and a result of the second estimation; and receive afirst finalization instruction to finalize an inspection result to berecorded or output to an external apparatus based on sequentiallyacquired inspection results of the subject, and a second finalizationinstruction to finalize inspection region information to be recorded oroutput to the external apparatus in association with the inspectionresult, wherein the second finalization instruction for the inspectionregion information is received before the first finalization instructionfor the inspection result.